自我监督学习中的最新作品通过以对象为中心或基于区域的对应目标进行预处理，在场景级密集的预测任务上表现出了强劲的表现。在本文中，我们介绍了区域对象表示学习（R2O），该学习统一了基于区域的和以对象为中心的预处理。 R2O通过训练编码器以动态完善基于区域的段为中心的蒙版，然后共同学习掩模中内容的表示形式。 R2O使用“区域改进模块”将使用区域级先验生成的小图像区域分组为较大的区域，这些区域倾向于通过聚类区域级特征对应对应对象。随着训练的进展，R2O遵循了一个区域到对象的课程，该课程鼓励学习区域级的早期特征并逐渐进步以训练以对象为中心的表示。使用R2O的表示形式导致了Pascal VOC（+0.7 MIOU）和CityScapes（+0.4 MIOU）的语义细分表现最先进的表现，并在MS Coco（+0.3 Mask AP）上进行了实例细分。此外，在对Imagenet进行了预审进之后，R2O预处理的模型能够超过Caltech-UCSD Birds 200-2011数据集（+2.9 MIOU）的无监督物体细分中现有的最新对象细分。我们在https://github.com/kkallidromitis/r2o上提供了这项工作的代码/模型。

尽管腿部机器人的运动计划表现出了巨大的成功，但具有灵活的多指抓握的腿部机器人的运动计划尚未成熟。我们提出了一个有效的运动计划框架，用于同时解决运动（例如，质心动力学），抓地力（例如，贴片接触）和触点（例如步态）问题。为了加速计划过程，我们建议基于乘数的交替方向方法（ADMM）提出分布式优化框架，以求解原始的大型混合构成非整数非线性编程（MINLP）。最终的框架使用混合构成二次编程（MIQP）来求解联系人和非线性编程（NLP）来求解非线性动力学，这些动力学在计算方面更可行，对参数较不敏感。此外，我们通过微蜘蛛抓手从极限表面明确执行补丁接触约束。我们在硬件实验中演示了我们提出的框架，这表明多限制机器人能够实现各种动作，包括在斜坡角度45 {\ deg}的情况下进行较短的计划时间。

本文介绍了Scalucs，这是一种四足动物，该机器人在地上，悬垂和天花板上爬上攀爬，并在地面上爬行。 Scaleer是最早的自由度四束机器人之一，可以在地球的重力下自由攀爬，也是地面上最有效的四足动物之一。在其他最先进的登山者专门攀登自己的地方，Scaleer承诺使用有效载荷\ Textit {和}地面运动实践自由攀爬，这实现了真正的多功能移动性。新的攀登步态滑冰步态通过利用缩放器的身体连锁机制来增加有效载荷。 Scaleer在地面上达到了最大归一化的运动速度，即$ 1.87 $ /s，$ 0.56 $ m /s，$ 1.2 $ /min，或$ 0.42 $ m /min /min的岩石墙攀爬。有效载荷能力达到地面上缩放器重量的233美元，垂直墙上的$ 35 $％。我们的山羊抓手是一种机械适应的两指抓手，成功地抓住了凸凸和非凸的对象，并支持缩放器。

Agents that can follow language instructions are expected to be useful in a variety of situations such as navigation. However, training neural network-based agents requires numerous paired trajectories and languages. This paper proposes using multimodal generative models for semi-supervised learning in the instruction following tasks. The models learn a shared representation of the paired data, and enable semi-supervised learning by reconstructing unpaired data through the representation. Key challenges in applying the models to sequence-to-sequence tasks including instruction following are learning a shared representation of variable-length mulitimodal data and incorporating attention mechanisms. To address the problems, this paper proposes a novel network architecture to absorb the difference in the sequence lengths of the multimodal data. In addition, to further improve the performance, this paper shows how to incorporate the generative model-based approach with an existing semi-supervised method called a speaker-follower model, and proposes a regularization term that improves inference using unpaired trajectories. Experiments on BabyAI and Room-to-Room (R2R) environments show that the proposed method improves the performance of instruction following by leveraging unpaired data, and improves the performance of the speaker-follower model by 2\% to 4\% in R2R.

A polarization camera has great potential for 3D reconstruction since the angle of polarization (AoP) and the degree of polarization (DoP) of reflected light are related to an object's surface normal. In this paper, we propose a novel 3D reconstruction method called Polarimetric Multi-View Inverse Rendering (Polarimetric MVIR) that effectively exploits geometric, photometric, and polarimetric cues extracted from input multi-view color-polarization images. We first estimate camera poses and an initial 3D model by geometric reconstruction with a standard structure-from-motion and multi-view stereo pipeline. We then refine the initial model by optimizing photometric rendering errors and polarimetric errors using multi-view RGB, AoP, and DoP images, where we propose a novel polarimetric cost function that enables an effective constraint on the estimated surface normal of each vertex, while considering four possible ambiguous azimuth angles revealed from the AoP measurement. The weight for the polarimetric cost is effectively determined based on the DoP measurement, which is regarded as the reliability of polarimetric information. Experimental results using both synthetic and real data demonstrate that our Polarimetric MVIR can reconstruct a detailed 3D shape without assuming a specific surface material and lighting condition.

Text-to-speech synthesis (TTS) is a task to convert texts into speech. Two of the factors that have been driving TTS are the advancements of probabilistic models and latent representation learning. We propose a TTS method based on latent variable conversion using a diffusion probabilistic model and the variational autoencoder (VAE). In our TTS method, we use a waveform model based on VAE, a diffusion model that predicts the distribution of latent variables in the waveform model from texts, and an alignment model that learns alignments between the text and speech latent sequences. Our method integrates diffusion with VAE by modeling both mean and variance parameters with diffusion, where the target distribution is determined by approximation from VAE. This latent variable conversion framework potentially enables us to flexibly incorporate various latent feature extractors. Our experiments show that our method is robust to linguistic labels with poor orthography and alignment errors.

End-to-end text-to-speech synthesis (TTS) can generate highly natural synthetic speech from raw text. However, rendering the correct pitch accents is still a challenging problem for end-to-end TTS. To tackle the challenge of rendering correct pitch accent in Japanese end-to-end TTS, we adopt PnG~BERT, a self-supervised pretrained model in the character and phoneme domain for TTS. We investigate the effects of features captured by PnG~BERT on Japanese TTS by modifying the fine-tuning condition to determine the conditions helpful inferring pitch accents. We manipulate content of PnG~BERT features from being text-oriented to speech-oriented by changing the number of fine-tuned layers during TTS. In addition, we teach PnG~BERT pitch accent information by fine-tuning with tone prediction as an additional downstream task. Our experimental results show that the features of PnG~BERT captured by pretraining contain information helpful inferring pitch accent, and PnG~BERT outperforms baseline Tacotron on accent correctness in a listening test.

Edema is a common symptom of kidney disease, and quantitative measurement of edema is desired. This paper presents a method to estimate the degree of edema from facial images taken before and after dialysis of renal failure patients. As tasks to estimate the degree of edema, we perform pre- and post-dialysis classification and body weight prediction. We develop a multi-patient pre-training framework for acquiring knowledge of edema and transfer the pre-trained model to a model for each patient. For effective pre-training, we propose a novel contrastive representation learning, called weight-aware supervised momentum contrast (WeightSupMoCo). WeightSupMoCo aims to make feature representations of facial images closer in similarity of patient weight when the pre- and post-dialysis labels are the same. Experimental results show that our pre-training approach improves the accuracy of pre- and post-dialysis classification by 15.1% and reduces the mean absolute error of weight prediction by 0.243 kg compared with training from scratch. The proposed method accurately estimate the degree of edema from facial images; our edema estimation system could thus be beneficial to dialysis patients.

Peripheral blood oxygen saturation (SpO2), an indicator of oxygen levels in the blood, is one of the most important physiological parameters. Although SpO2 is usually measured using a pulse oximeter, non-contact SpO2 estimation methods from facial or hand videos have been attracting attention in recent years. In this paper, we propose an SpO2 estimation method from facial videos based on convolutional neural networks (CNN). Our method constructs CNN models that consider the direct current (DC) and alternating current (AC) components extracted from the RGB signals of facial videos, which are important in the principle of SpO2 estimation. Specifically, we extract the DC and AC components from the spatio-temporal map using filtering processes and train CNN models to predict SpO2 from these components. We also propose an end-to-end model that predicts SpO2 directly from the spatio-temporal map by extracting the DC and AC components via convolutional layers. Experiments using facial videos and SpO2 data from 50 subjects demonstrate that the proposed method achieves a better estimation performance than current state-of-the-art SpO2 estimation methods.

The modern dynamic and heterogeneous network brings differential environments with respective state transition probability to agents, which leads to the local strategy trap problem of traditional federated reinforcement learning (FRL) based network optimization algorithm. To solve this problem, we propose a novel Differentiated Federated Reinforcement Learning (DFRL), which evolves the global policy model integration and local inference with the global policy model in traditional FRL to a collaborative learning process with parallel global trends learning and differential local policy model learning. In the DFRL, the local policy learning model is adaptively updated with the global trends model and local environment and achieves better differentiated adaptation. We evaluate the outperformance of the proposal compared with the state-of-the-art FRL in a classical CartPole game with heterogeneous environments. Furthermore, we implement the proposal in the heterogeneous Space-air-ground Integrated Network (SAGIN) for the classical traffic offloading problem in network. The simulation result shows that the proposal shows better global performance and fairness than baselines in terms of throughput, delay, and packet drop rate.